[VL] Enable native Parquet write for complex types (Struct/Array/Map) by Zouxxyy · Pull Request #11788 · apache/gluten

Zouxxyy · 2026-03-19T03:50:34Z

What changes are proposed in this pull request?

Enable native Parquet write for complex types (Struct/Array/Map) in Velox backend.

Velox's parquet writer converts vectors to Arrow then writes via Arrow's Parquet writer, which natively supports nested types. The previous Scala-side type restrictions were unnecessary.

Changes:

Remove supportNativeWrite gate — no longer needed since supportWriteFilesExec handles validation
Remove Struct/Array/Map restrictions from validateDataTypes for Parquet (only YearMonthIntervalType remains blocked, as Arrow has no mapping for it)
Make validateDataTypes recursively check nested types for YearMonthIntervalType
Add tests for struct, array, map, and nested struct writes

Note:

Exclude SPARK-47546 variant binary tests (Spark 4.0/4.1). Velox parquet writer writes all struct fields as OPTIONAL, but variant type requires REQUIRED. This is a Velox-side limitation (exportToArrow unconditionally sets nullable flag). Fix needs Velox changes, out of scope for this PR.

How was this patch tested?

New tests in VeloxParquetWriteSuite. Existing tests pass.

Was this patch authored or co-authored using generative AI tooling?

Cooperated with: Kiro (Claude Opus 4.6)

github-actions · 2026-03-19T03:51:04Z

Run Gluten Clickhouse CI on x86

Zouxxyy · 2026-03-19T03:53:11Z

Write Validation Chain (spark3.4+) Generated-by: Kiro (Claude Opus 4.6)

 +-------------------------------------------------------------+
 |              Spark SQL Write Statement                        |
 |  (INSERT INTO / CTAS / INSERT OVERWRITE DIR / Hive INSERT)   |
 +-----------------------------+-------------------------------+
                               |
                               v
 +-----------------------------+-------------------------------+
 |  DataWritingCommandExec(cmd, child)                         |
 |    child contains WriteFilesExec (Spark 3.4+)               |
 +-----------------------------+-------------------------------+
                               |
 ==============================|====================================
  Gate 1: OffloadSingleNodeRules (standard columnar offload)
 ==============================|====================================
                               |
                               v
              +--------------------------------------+
              |  WriteFilesExec matched              |
              |  -> WriteFilesExecTransformer        |
              |  -> ColumnarWriteFilesExec           |
              +------+-------------------------------+
                     |
 ====================|=============================================
  Gate 2: doValidateInternal  (WriteFilesExecTransformer.scala)
 ====================|=============================================
                     |
                     v
              +--------------------------------------+
              |  2a. Constant complex type check     |
              |                                      |
              |  Project contains                    |
              |  Literal(ArrayType|MapType)?          |
              +------+----------------+--------------+
                 YES |                | NO
                     v                v
               +--------+   +-------------------------------+
               |FALLBACK|   | 2b. supportWriteFilesExec()   |
               +--------+   +-------------------------------+
                                     |
                                     v
                    +---------------------------------------------------+
                    |   Validation chain (in order, any fail = fallback) |
                    |                                                   |
                    |  (1) validateFileFormat                            |
                    |      OK ParquetFileFormat                         |
                    |      OK HiveFileFormat (Parquet SerDe)            |
                    |      X  Other formats                             |
                    |                    |                               |
                    |                    v                               |
                    |  (2) validateCompressionCodec                      |
                    |      X  brotli / lzo / lz4raw / lz4_raw           |
                    |                    |                               |
                    |                    v                               |
                    |  (3) validateFieldMetadata                         |
                    |      X  StructField.metadata != Metadata.empty    |
                    |                    |                               |
                    |                    v                               |
                    |  (4) validateDataTypes (recursive)                 |
                    |      X  YearMonthIntervalType                     |
                    |         (including nested in Struct/Array/Map)     |
                    |      OK StructType / ArrayType / MapType          |
                    |      OK all other primitive types                  |
                    |                    |                               |
                    |                    v                               |
                    |  (5) validateWriteFilesOptions                     |
                    |      X  maxRecordsPerFile > 0                     |
                    |                    |                               |
                    |                    v                               |
                    |  (6) validateBucketSpec                            |
                    |      X  Non Hive-compatible bucket write          |
                    +------------------------+--------------------------+
                                             |
                                      +------+------+
                                      | All passed? |
                                      +--+-------+--+
                                      NO |       | YES
                                         v       v
                                   +--------+   +------------------------------+
                                   |FALLBACK|   | 2c. doNativeValidation()     |
                                   +--------+   |  Substrait -> C++ validation |
                                                +---------------+--------------+
                                                                |
 ===============================================================|=========
  Gate 3: C++ Validator  (SubstraitToVeloxPlanValidator.cc)
 ===============================================================|=========
                                                                |
                                                                v
                              +--------------------------------------+
                              |  validate(WriteRel)                  |
                              |                                      |
                              |  3a. Recursively validate input plan |
                              |  3b. Parse input row type            |
                              |  3c. Validate partition column types:|
                              |      OK BOOLEAN / TINYINT / SMALLINT |
                              |      OK INTEGER / BIGINT             |
                              |      OK VARCHAR / VARBINARY          |
                              |      X  Other types as partition col |
                              |                                      |
                              |  Data columns: no type restriction   |
                              +------+----------------+--------------+
                                  NO |                | YES
                                     v                v
                               +--------+   +---------------------+
                               |FALLBACK|   | OK: Native Write    |
                               +--------+   +---------+-----------+
                                                      |
                                                      v
                              +--------------------------------------+
                              |           Execution Layer             |
                              |                                      |
                              |  VeloxColumnarWriteFilesExec          |
                              |       |                              |
                              |       v                              |
                              |  VeloxParquetDataSource               |
                              |       |                              |
                              |       v                              |
                              |  velox::parquet::Writer              |
                              |  (backed by Arrow Parquet Writer)    |
                              +--------------------------------------+

Copilot

Pull request overview

This PR enables Velox’s native Parquet write path to support complex Spark SQL types (Struct/Array/Map) by removing earlier type-gating and adjusting the Velox write validation to allow nested types for Parquet.

Changes:

Removes the schema-based “native write supported” gate (supportNativeWrite) and relies on the WriteFiles validation path instead.
Updates Velox write validation to allow Parquet StructType (still blocks YearMonthIntervalType) and reorders validation checks.
Simplifies Delta Parquet native-writability checks and adds new Velox Parquet write tests for complex/nested types.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/datasources/GlutenWriterColumnarRules.scala	Removes schema gate before enabling native write properties/adaptor injection.
gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/BackendSettingsApi.scala	Deletes `supportNativeWrite` from the backend settings API.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala	Allows Parquet struct types; refactors/reorders native write validation chain.
backends-velox/src/test/scala/org/apache/spark/sql/execution/VeloxParquetWriteSuite.scala	Adds native Parquet write coverage for struct/array/map and nested struct.
backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/files/GlutenDeltaFileFormatWriter.scala	Removes dependency on deleted Parquet companion helper; forces native-writable flag.
backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/GlutenParquetFileFormat.scala	Removes fallback branch + companion object; always uses Gluten Parquet OutputWriterFactory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Zouxxyy · 2026-03-20T09:37:39Z

...it/src/main/scala/org/apache/spark/sql/execution/datasources/GlutenWriterColumnarRules.scala

      case rc @ DataWritingCommandExec(cmd, child) =>
        // The same thread can set these properties in the last query submission.
        val format =
-          if (
-            BackendsApiManager.getSettings.supportNativeWrite(child.schema.fields) &&
-            BackendsApiManager.getSettings.enableNativeWriteFiles()
-          ) {
+          if (BackendsApiManager.getSettings.enableNativeWriteFiles()) {
            getNativeFormat(cmd)
          } else {


Not a real issue. On Spark 3.4+, NativeWritePostRule only injects FakeRowAdaptor — the actual native/fallback decision is made by WriteFilesExecTransformer.doValidateInternal, which already rejects constant complex types. When validation fails, WriteFilesExec stays vanilla and FakeRowAdaptor is a harmless pass-through.

Also, #11787 (already merged) restricts NativeWritePostRule to Spark 3.3 only, so this code path is no longer reachable on 3.4+.

backends-velox/src-delta33/main/scala/org/apache/spark/sql/delta/GlutenParquetFileFormat.scala

github-actions · 2026-03-19T17:21:39Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-20T00:29:19Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-20T00:30:26Z

Run Gluten Clickhouse CI on x86

github-actions · 2026-03-20T09:31:41Z

Run Gluten Clickhouse CI on x86

Zouxxyy · 2026-03-20T11:54:32Z

CC @jackylee-ch @jinchengchenghh @zhztheplayer for a look, thanks.

For reference, the Iceberg write path does not restrict complex types (Struct/Array/Map) at the format level too.

jinchengchenghh · 2026-03-20T12:19:45Z

gluten-ut/spark40/src/test/scala/org/apache/gluten/utils/velox/VeloxTestSettings.scala

  enableSuite[GlutenVariantSuite]
+    // TODO: Velox parquet writer marks all struct fields as OPTIONAL (nullable),
+    //  but Spark's variant type requires REQUIRED fields. Needs Velox-side fix.
+    .exclude("SPARK-47546: invalid variant binary")


Please add it to #11088

It appears I don't have permission to edit this. Perhaps I can modify it in the future—I actually have experience working with variants.

Could you add sub issue to it?

Added #11803

Thanks! LGTM

jinchengchenghh

Thanks!

jackylee-ch

Basically looks good to me. Left few comments

backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala

jackylee-ch · 2026-03-20T12:33:02Z

backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala

-              field.dataType match {
-                case _: StructType => Some("StructType")
-                case _: ArrayType => Some("ArrayType")
-                case _: MapType => Some("MapType")


Do we have any native test cases for HiveFileFormat's StructType, ArrayType, and MapType?

thanks for the review, add the test

github-actions · 2026-03-20T13:21:38Z

Run Gluten Clickhouse CI on x86

github-actions bot added CORE works for Gluten Core VELOX labels Mar 19, 2026

jackylee-ch requested a review from Copilot March 19, 2026 04:02

Copilot AI reviewed Mar 19, 2026

View reviewed changes

Zouxxyy added 3 commits March 20, 2026 08:29

native write

1514ade

fix test

ac1b295

format

3d5d937

Zouxxyy force-pushed the dev/native-write branch from ce64e75 to 3d5d937 Compare March 20, 2026 00:29

Zouxxyy changed the title ~~[VL] Support native Parquet write for complex types (Struct/Array/Map)~~ [VL] Enable native Parquet write for complex types (Struct/Array/Map) Mar 20, 2026

ignore 2 case

3ca3025

jinchengchenghh reviewed Mar 20, 2026

View reviewed changes

jinchengchenghh approved these changes Mar 20, 2026

View reviewed changes

jackylee-ch reviewed Mar 20, 2026

View reviewed changes

for comment

cede51b

Conversation

Zouxxyy commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

github-actions bot commented Mar 19, 2026

Uh oh!

Zouxxyy commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Mar 19, 2026

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

Zouxxyy commented Mar 20, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jinchengchenghh left a comment

Choose a reason for hiding this comment

Uh oh!

jackylee-ch left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Zouxxyy commented Mar 19, 2026 •

edited

Loading

Zouxxyy commented Mar 19, 2026 •

edited

Loading